Situation-aware message presentation for automotive messaging

ABSTRACT

A text message processing arrangement is described for use in a mobile environment. A mobile messaging application processes user text messages during a user messaging session. A user state model reflects situational parameters to characterize user cognitive load. A functionality control module adjusts functional performance of the mobile messaging application based on the user state model.

TECHNICAL FIELD

The present invention relates to natural language understandingtechnology, and more specifically to situation-aware message handling ina mobile messaging application.

BACKGROUND ART

Instant messaging (IM) has become part of everyday life and its usersrequire IM availability in many different situations. IM is commonlymediated through a broad range of devices including computers, tablets,and cell phones. However, these devices are not very appropriate forsome mobile environments such as in-car use. Their handling requiressignificant attention from the user, which can result in dangerousdriver distraction. While in some driving situations it may be dangerousto perform any extra tasks such as instant messaging, there also aresituations when it can be relatively safe.

SUMMARY

Embodiments of the present invention are directed to a text messageprocessing arrangement for use in a mobile environment such as thepassenger compartment of an automobile. A mobile messaging applicationsuch as an instant messaging application processes user text messagesduring a user messaging session. A user state model reflects situationalparameters to characterize user cognitive load. A functionality controlmodule adjusts functional performance of the mobile messagingapplication based on the user state model.

The functionality control module may adjust functional performance ofthe mobile messaging application using a safe/dangerous classificationbased on the user state model. The functional performance which isadjusted may include at least one of a message summarizingfunctionality, a messaging sequencing control functionality, initiatingpresentation of a new message, continuing presentation of a currentmessage, and form of user interaction such as: speech interaction, textinteraction, and interaction speed. And the mobile messaging applicationmay further provide a user status update output indicating the currentstatus of a system user based on the user state model.

The situational parameters may include some number of recent events,parameters characterizing the passenger compartment environment, and/orparameters characterizing an automobile driving environment such as:automobile speed, approaching objects, traffic, conditions, roadconditions, weather conditions, and physical environment conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a text message processing system according to oneembodiment of the present invention.

FIG. 2 shows various functional steps of situation aware processing oftext messages according to an embodiment.

DETAILED DESCRIPTION

Embodiments of the present invention are directed to a text messagingsystem for use in a mobile environment such as automotive messaging.Instant messaging, emailing, and/or social media messaging (e.g.,Twitter, LinkedIn, Facebook etc.) and possibly other informationprocessing components are integrated into a single system andhuman-machine interface using speech inputs and/or outputs. For suchautomotive applications the system takes into account the drivingsituation and adapts its mechanisms and functionality so that the usercognitive load is always safe. The behavior of the system is adapteddepending on the state of the driver, the situation in the car, and thedriving situation. Other parties can be notified of the user's currentsituation to allow them to consider how much attention they request fromhim. For example, a phrase can be added to the first chat message of agiven dialog: “user X is currently driving in high-speed traffic anddictating messages with Dragon Dictation software by Nuance.”

FIG. 1 shows a text message processing system 100 and FIG. 2 showsvarious functional steps in situation aware processing of text messagesaccording to one embodiment of the present invention. During a usermessaging session, a mobile messaging application 101 processes at leastone user text message, step 201, received by wireless link 102 from acommunications network 103 connected to other text messaging systems.For example, the mobile messaging application 101 may perform speechprocessing of the text messages such as by automatic speech recognition(ASR) and/or text-to-speech (TTS) speech synthesis. A user state model104 characterizes user cognitive load, step 202, accumulating situationclues and building a model of the user cognitive state based onsituational parameters such as some number of recent events, parameterscharacterizing the passenger compartment environment, and/or parameterscharacterizing an automobile driving environment such as: automobilespeed, approaching objects, traffic, conditions, road conditions,weather conditions, and physical environment conditions.

The user state model 104 can collect information about the current roadsituation by observing clues from variety of sensors such as:

-   -   Sensors indicating distance from closest obstacle in front of        car; speed of approaching the closest obstacle.    -   Sensors indicating discrepancy between direction of the car and        direction of the road.    -   GPS and geographic information providing the information about        quality of road, closest intersection, profile of the road.    -   Detection of overtaking based on clues from GPS, lane tracking        subsystem, time evidence.    -   Sensors of other activities in a car—manual and automatic gear        shifting, controlling radio, air-conditioning . . . .    -   Detection of human conversations in a car.    -   Rain sensors, windshield wipers activity, or temperature        outside.    -   Car speed with respect to road type.    -   Stress or fatigue detection based on analyses of voice, face, or        additional biometric information about the driver.        Applicant notes that some sensor situational parameters are        independent of user action, such as rain detection, and some are        dependent upon user action, such as vehicle speed.

A functionality control module 105 then adjusts functional performanceof the mobile messaging application, step 203, based on the user statemodel 104, for example, by making a safe/dangerous classification. Thefunctional performance adjusted by the functionality control module 105in step 203 may include at least one of a message summarizingfunctionality, a messaging sequencing control functionality, initiatingpresentation of a new message, continuing presentation of a currentmessage, and form of user interaction such as: speech interaction, textinteraction, and interaction speed.

For example, the functionality control module 105 may classify thecurrent driving situation and make functionality decisions such as:

-   -   (1) Is the information allowed to play?    -   (2) Should the current playback be allowed to complete or should        it be interrupted?    -   (3) How the message should be rendered (messages        playback/beep/graphical display)?    -   (4) What should be the prosody and speed of the playback?        Informational data such as from GPS+map+destination can be used        to plan system intervention/adjustments, for example, some or        all of the current messages may not be played back when the        vehicle is close to a busy difficult intersection or the        simultaneous (to audio playback) graphical display may be used        only when the car is parked or at low speed.

The user state model 104 and/or the functionality control module 105 canempirically weight the sensor data and combine their values to one ormore user distraction measures such as a multi-dimensional distractionvector. A set of rules and the values of the distraction vector can beused to determine how to handle of a system announcement (playback ofincoming message, status change report). Distraction measures also canreflect delayed impacts of certain events. For example, shortly aftergear shifting the user is likely to be occupied by handling the impactof the shift (car acts differently).

When the distraction vector indicates mild to moderate user distraction(i.e., moderate cognitive load) message playback may be delayed (e.g.,by a few seconds) in expectation of an upcoming better cognitivesituation. Previously played messages may be uninterrupted while newplaybacks may be postponed. In addition or alternatively, messageplayback may be replaced by shorter messages such as a beep thatannounces arrival of a new message and/or playback of a shorterinformational announcement such as: “message from John.” Lower prioritymessages may not be played back at all. As the cognitive load anddistraction level increases higher, the system functionality may bealtered accordingly. For example, all message playbacks may be stoppedimmediately. The user may be able to control how much the systemintervenes in the natural flow of the conversation (message processing)by setting one or more operating thresholds.

The mobile messaging application 101 may also provide a user statusupdate output over the wireless link 102 to other users in thecommunications network 103 to indicate to them the current status of asystem user based on the user state model 104. User status can be setautomatically based on the accumulated situational clues. For example auser status “incar busy” can indicate that user is busy by handling thecurrent driving situation. Other user status message examples includewithout limitation: in car, busy in car, bored in car, stuck in traffic,on the phone in the car, chatting in car, etc. User status may bereported automatically based on the driving situation (i.e., from theuser state model 104). In addition or alternatively, user status outputsmay be set manually by voice commands from the user. Examples of relatedvoice commands include:

USER: (Stop|Start) reporting status

USER: Set status to (busy|offline| . . . )

Similarly, the message processing system 100 may inform the user mayabout the status of other messaging parties (e.g., in a chatconversation and/or from a user contact list) based on the status of theuser state model 104, for example, by voice only when the drivingcognitive load is low enough. The user may be able to restrictnotifications or completely switch them off. Status messages can bereported based on driving conditions, for example when user is busy whenturning at an intersection, announcements may be postponed. Examples ofrelated voice commands and notifications include:

-   -   SYSTEM: User X (became active|is in a car|is busy| . . . )    -   USER: Report status change of X and Y    -   USER: What is the status of X?    -   SYSTEM: X is offline    -   USER: Who is active from group Z?    -   SYSTEM: Peter and John are active in group Z

Text messages can pop up asynchronously and so the playback of themmight overlap. Thus the message processing system 100 may include aserialization module to ensure ordered playback of text messages.Serialization strategies may group together messages from a singleconversation. Message serialization may be prioritized so that messagesbelonging to higher priority users are played back earlier. Brief systemannouncements of incoming text messages may be played instead of fullmessage texts. Messages from users not in focus may be replaced by anearcon or a counterpart name playback announcing message arrival(without playing the message text back). Message playback may becontrolled based on driving situation (GPS+map info, sensors, steeringwheel activity, and turn signals). The serialization strategies may becustomizable by system software and/or manually by the user. The usermay be able to later request by voice command a full version of anyshortened messages. Examples of related voice commands include:

-   -   USER: What happened?    -   SYSTEM: Martin logged off Message from John: Hello    -   USER: (List|Play) new messages. (Referring to shortened messages        not played back yet or played in shortened form)

Embodiments of the invention may be implemented in whole or in part inany conventional computer programming language. For example, preferredembodiments may be implemented in a procedural programming language(e.g., “C”) or an object oriented programming language (e.g., “C++”,Python). Alternative embodiments of the invention may be implemented aspre-programmed hardware elements, other related components, or as acombination of hardware and software components.

Embodiments can be implemented in whole or in part as a computer programproduct for use with a computer system. Such implementation may includea series of computer instructions, which are executable by a processor,fixed either on a tangible medium, such as a computer readable medium(e.g., a diskette, CD-ROM, ROM, or fixed disk) or transmittable to acomputer system (e.g., a computer including processor), via a modem orother interface device, such as a communications adapter connected to anetwork over a medium. The medium may be either a tangible medium (e.g.,optical or analog communications lines) or a medium implemented withwireless techniques (e.g., microwave, infrared or other transmissiontechniques). The series of computer instructions embodies all or part ofthe functionality previously described herein with respect to thesystem. Those skilled in the art should appreciate that such computerinstructions can be written in a number of programming languages for usewith many computer architectures or operating systems. Furthermore, suchinstructions may be stored in any memory device, such as semiconductor,magnetic, optical or other memory devices, and may be transmitted usingany communications technology, such as optical, infrared, microwave, orother transmission technologies. It is expected that such a computerprogram product may be distributed as a removable medium withaccompanying printed or electronic documentation (e.g., shrink wrappedsoftware), preloaded with a computer system (e.g., on system ROM orfixed disk), or distributed from a server or electronic bulletin boardover the network (e.g., the Internet or World Wide Web). Of course, someembodiments of the invention may be implemented as a combination of bothsoftware (e.g., a computer program product) and hardware. Still otherembodiments of the invention are implemented as entirely hardware, orentirely software (e.g., a computer program product).

Although various exemplary embodiments of the invention have beendisclosed, it should be apparent to those skilled in the art thatvarious changes and modifications can be made which will achieve some ofthe advantages of the invention without departing from the true scope ofthe invention.

What is claimed is:
 1. A text message processing system for use in amobile environment, the system comprising: a mobile messagingapplication for processing user text messages during a user messagingsession; a user state model characterizing a cognitive load of the userbased upon situational parameters including vehicle sensor informationfor a vehicle driven by a user, wherein the situational parameters aredependent upon user action, wherein the situational parameters includepassenger compartment environment and vehicle speed, wherein the systemdetermines, using a processor, a plurality of distraction levels fromthe situational parameters, wherein each of the plurality of distractionlevels corresponds to the cognitive load of the user, wherein a firstone of the distraction levels is based upon a first one of thesituational parameters corresponding to the passenger compartmentenvironment and a second one of the distraction levels is based upon thevehicle speed; and a functionality control module for adjustingfunctional performance of the mobile messaging application based on thedistraction levels, such that a first one of the plurality ofdistraction levels corresponds to no message playback at all and asecond one of the plurality of distraction levels corresponds to notplaying back a message having a low priority.
 2. A system according toclaim 1, wherein the situational parameters include number of recentevents.
 3. A system according to claim 1, wherein the mobile environmentis the passenger compartment of an automobile.
 4. A system according toclaim 3, wherein the situational parameters include one or moreparameters characterizing a passenger compartment environment.
 5. Asystem according to claim 3, wherein the situational parameters includeone or more parameters characterizing an automobile driving environment.6. A system according to claim 5, wherein the situational parametersinclude one or more parameters characterizing approaching objects,traffic conditions, road conditions, weather conditions, and physicalenvironment conditions.
 7. A system according to claim 1, wherein thefunctional performance includes at least one of initiating presentationof a new message, continuing presentation of a current message, and formof user interaction.
 8. A system according to claim 7, wherein the formof user interaction includes at least one of speech interaction, textinteraction, and interaction speed.
 9. A system according to claim 1,wherein the functional performance includes a message summarizingfunctionality.
 10. A system according to claim 1, wherein the functionalperformance includes a messaging sequencing control functionality.
 11. Asystem according to claim 1, wherein the mobile messaging applicationfurther provides a user status update output indicating the currentstatus of a system user based on the user state model.
 12. A systemaccording to claim 1, wherein the mobile messaging application is aninstant messaging application.
 13. The method according to claim 1,further including generating a status update for the user correspondingto the distraction level of the user for transmission to other users viaa wireless link.
 14. The method according to claim 13, wherein thestatus update comprises one or more of in car busy, in car, stuck intraffic, and/or on phone.
 15. The method according to claim 13, furtherincluding receiving a status for other messaging parties.
 16. The methodaccording to claim 13, further including postponing messages for thestatus for other messaging parties based upon driver conditions for theuser.
 17. A method of processing text messages in a mobile environment,the method comprising: processing user text messages in a user messagingsession with a mobile messaging application; characterizing a usercognitive load with a user state model based upon situationalparameters, including vehicle sensor information for a vehicle driven bya user, wherein the situational parameters are dependent upon useraction, wherein the situational parameters include passenger compartmentenvironment and vehicle speed, determining a plurality of distractionlevels from the situational parameters, wherein each of the plurality ofdistraction levels corresponds to the cognitive load of the user;wherein a first one of the distraction levels is based upon a first oneof the situational parameters corresponding to the passenger compartmentenvironment and a second one of the distraction levels is based upon thevehicle speed; and adjusting functional performance of the mobilemessaging application based on the distraction levels, such that a firstone of the plurality of distraction levels corresponds to no messageplayback at all and a second one of the plurality of distraction levelscorresponds to not playing back a message having a low priority.
 18. Amethod according to claim 17, wherein the situational parameters includenumber of recent events.
 19. A method according to claim 17, wherein themobile environment is the passenger compartment of an automobile.
 20. Amethod according to claim 19, wherein the situational parameters includeone or more parameters characterizing a passenger compartmentenvironment.
 21. A method according to claim 19, wherein the situationalparameters include one or more parameters characterizing an automobiledriving environment.
 22. A method according to claim 21, wherein thesituational parameters include one or more parameters characterizingautomobile speed, approaching objects, traffic, conditions, roadconditions, weather conditions, and physical environment conditions. 23.A method according to claim 17, wherein the functional performanceincludes at least one of initiating presentation of a new message,continuing presentation of a current message, and form of userinteraction.
 24. A method according to claim 23, wherein the form ofuser interaction includes at least one of speech interaction, textinteraction, and interaction speed.
 25. A method according to claim 17,wherein the functional performance includes a message summarizingfunctionality.
 26. A method according to claim 17, wherein thefunctional performance includes a messaging sequencing controlfunctionality.
 27. A method according to claim 17, wherein the mobilemessaging application further provides a user status update outputindicating the current status of a method user based on the user statemodel.
 28. A method according to claim 17, wherein the mobile messagingapplication is an instant messaging application.
 29. A computer programproduct, comprising: a non-transitory computer readable medium havingstored instructions that enable a machine to: process text messages in amobile environment by processing user text messages in a user messagingsession with a mobile messaging application; characterize a usercognitive load with a user state model based upon situationalparameters, including vehicle sensor information for a vehicle driven bya user, wherein the situational parameters are dependent upon useraction, wherein the situational parameters include passenger compartmentenvironment and vehicle speed, determine a plurality of distractionlevels from the situational parameters, wherein each of the plurality ofdistraction levels corresponds to the cognitive load of the user,wherein a first one of the distraction levels is based upon a first oneof the situational parameters corresponding to the passenger compartmentenvironment and a second one of the distraction levels is based upon thevehicle speed; and adjust functional performance of the mobile messagingapplication based on the distraction levels, such that a first one ofthe plurality of distraction levels corresponds to no message playbackat all and a second one of the plurality of distraction levelscorresponds to not playing back a message having a low priority.